This application relates to image processing in digital cameras and other electronic digital image acquisition devices, and particularly to techniques of improving noise reduction techniques for such images.
FIG. 1 is a block diagram of some elements of a typical imaging system, such as a digital still or video camera. The image is captured at block 101, which schematically represents the various front-end elements (lens, sensor array, analog to digital conversion, etc.) that precede the Image Processing Unit (IPU) 111 and supply the image to the IPU. The image data can be supplied to the IPU as raw image, RGB, YCbCg422, YCbCr444, or other format where it can be received in a SDRAM 113, for example. The IPU will generally contain a number of additional elements, implemented in hardware, software, or some combination of these, as is familiar in the art, but which are suppressed here for simplification of the discussion. In this embodiment, the block labeled DRC (Dynamic Range Compensation) 117 in the middle of the processing chain receives a 3-color ×12 bit linear RGB image constructed by, in this embodiment, the Color Interpolation (CI) unit 115 if needed. (Although the 3-color ×12 bit linear RGB format is described here, other formats for the image data can also be used.) The RGB output image of the DRC block, here again taken as being a 3 color ×12 bit linear RGB image, is then processed by the rest of the Image Processing Unit (IPU) pipeline and, typically, eventually JPEG encoded.
Prior to the image data being received at the DRC block 117, it frequently undergoes a noise reduction process. The block labeled NR (Noise Reduction) 119 is interposed between the DRC element 117 and, here, the color interpolation unit 115. The NR block 119 receives the input pixel data Pin, subjects it to a noise reduction process, and outputs the result Pnr, which in turn is the input to DRC 117, whose pixel data output is labeled Pout. These algorithms are well known in the art and are well documented. When applying local Dynamic Range Compensation (DRC) on noisy images, that is, brightening dark areas of a noisy image, or enhancing contrast in bright areas of a noisy image, there is a side effect of varying noise levels in the image. This is due to the fact that DRC enhances noise as well as details. In effect, areas that went through DRC can appear much noisier than areas that did not go through DRC. Of course, if the NR unit 119 were to remove the noise completely, this would solve the problem; however, this would also remove much of the image detail and, in order to preserve texture and granularity in the image, it is common practice to reduce the amount of noise in the image, but not to remove it completely.
FIG. 2 is a block diagram of a noise reduction implementation. Pixels of an image, Pin(x,y), are stored in a memory buffer 213 and are accessed, using memory control 211 that supplies the read/write control signals and addressing, by a noise reduction unit 119 to perform the noise reduction algorithm. For simplicity, only two scales are used in this example, the current pixel and the region of its surrounding 8 pixels. The block 201 calculates the blending factor, α, based on the current pixel, Pin(x,y), and its surrounding neighbors. This is done by assessing the amount of detail in the region, setting the value α higher where detail is detected, and lower where uniformity is detected. This allows for noise reduction without too much loss of detail. The block 203 then averages the neighboring pixels:Pavg(x,y)=(Σ1−1Σ1−1Pin(x,y))/9where x and y are the column and row indices of the image and the sum in this embodiment is over the 3-by-3 pixel neighborhood centered oh the pixel at (x,y). Finally the current pixel is multiplied (at 205) by the blend factor (1−α) and added (at 209) to the average of its neighbors multiplied (at 207) by α. This procedure is repeated for each pixel in the image, thereby reducing the noise of the image.
Subsequently, the dynamic range of the image is enhanced as shown in the DRC block 117 in FIG. 3. The system computes a gain (or attenuation), G(x,y), for each sensor pixel at location (x,y). Each color component (Pnr) of the input RGB image (Rin, Gin, Bin) is multiplied by G(x,y) to produce the output color component for that pixel Pout(x,y):Pout(x,y)=G(x,y)*Pnr, where C=R, G or B and in the above equation, and following discussion, ‘*’ stands for multiplication.
As shown in FIG. 3, the color components of the input RGB image is received at block 301 where the grey intensity image, I(x,y), is determined as the maximum value of the three color components of the input image pixel (Rin, Gin, Bin):I(x,y)=max(Rin(x,y),Gin(x,y),Bin(x,y)).Other methods, such as those that use the luminance (Y), as in YcbCr or YUV, or other combination (e.g., Y=0.3R+0.6G+0.1B) could also be used.
In block 303, the log of I(x,y) is formed by means of a look-up table or by other computational means:L(x,y)=Log(I(x,y)).L(x,y) then serves as the measure of the image strength or amplitude that is input into block 305 where low pass filtering and non-linear combining are performed to produce an image, B(x,y), where details of this process in the basic embodiment are given with respect to FIG. 3. The image B(x,y) is applied to a gain look-up table, 307, to determine the gain values, G(x,y).
These various prior art methods tend to have a number of shortcomings when it comes to implementation in digital cameras, video, and other imaging systems. As described, previous implementations of digital cameras often reduce noise in an image by means of multi-scale algorithms, which blend each pixel with averages of the surrounding pixels at various scales, the degree of blending dependent on the current pixel and an assessment of the presence of detail in the region. Subsequently, the dynamic range of the image may be enhanced by increasing the gain of the image in areas of shadow, or reducing the gain of the image in regions of brightness. However, the concatenation of these two processes suffers from more apparent graininess and speckle in the regions compensated due to the fact that the dynamic range compensation enhances noise as well as details. Further, given that the development of sensors with higher pixel densities will tend to make pixels noisier, techniques to treat these problems will likely grow in importance.